Perspective correction for panoramic digital camera with remote processing

ABSTRACT

A method for converting a rectilinear image and a focal length into a cylindrical image parameterized by a height of the cylinder and an angular distance in a single buffer on a remote processing device. The method, on the remote processing device, comprising joining two or more images together to form a panoramic image with corrected perspective. The rectilinear transformation is “in-place” and requires only one buffer. Color correction and motion estimation is also carried out on the remote device. In an alternate embodiment, a computer readable medium corresponding to the above method is described.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application relates to technology similar to U.S. patentapplications Ser. No. 09/477,037, Ser. No. 09/477,036, Ser. No.09/476,652, now U.S. Pat. No. 6,456,323 Ser. No. 09/477,919, and Ser.No. 09/477,117, all being filed concurrently herewith and commonlyassigned herewith to STMicroelectronics Inc. and which are herebyincorporated by reference in their entirety hereinto.

PARTIAL WAIVER OF COPYRIGHT

All of the material in this patent application is subject to copyrightprotection under the copyright laws of the United States and of othercountries. As of the first effective filing date of the presentapplication, this material is protected as unpublished material.

However, permission to copy this material is hereby granted to theextent that the copyright owner has no objection to the facsimilereproduction by anyone of the patent documentation or patent disclosure,as it appears in the United States Patent and Trademark Office patentfile or records, but otherwise reserves all copyright rights whatsoever.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The invention disclosed broadly relates to the field of image processingand more specifically to image processing in a digital camera for takingpanoramic pictures.

2. Description of the Related Art

Today, panoramic photography is accomplished in various ways. One is touse a still camera mounted on a tripod to take a succession of shots asthe camera is pivoted around the tripod. In some cameras, a wider thanusual strip of film is exposed with special movable optics.

In other cameras, conventional format film, such as 35 mm film, ismasked during the exposure in the camera to provide a panoramic effect.The effect is panoramic but the whole exposure is limited by the fieldof view through the lens.

Other techniques for creating panoramic photography include tophysically cut and paste together strips of exposed film by carefullyaligning boundaries between edges of film.

The benefits of electronic photography have led to the growth of digitalcameras, that, unlike their film-based counterparts, store imagescaptured in memory into digital memory such as flash memory. To providepanoramic photography effects, these digital cameras can interface withpersonal computers for joining together two or more images into oneimage to provide a panoramic effect by joining edge boundaries ofimages. One such system is disclosed in U.S. Pat. No. 6,682,197, bynamed inventors Omid A. Moughadam, Stuart R. Ring, and John R. Squilla,entitled “Electronic Panoramic Camera For Use With An ExternalProcessor.”

Complicated panoramic digital cameras are available that rely onposition sensors or satellite communications for determining positioncoordinates. These position coordinates are used to help combine thepanoramic images. The process of combining scenes taken from differentcamera orientations is known as “Image Stitching.” One such system isdisclosed in U.S. Pat. No. 5,262,867 by named inventor Kiyonobu Kojimaentitled “Electronic Camera and Device for Panoramic Imaging and ObjectSearching” issued on Nov. 16, 1993.

A panoramic camera with a memory device for storing data from apreviously photographed portion of an object and a control device forenabling the display device to substantially display both the image tobe photographed and the image already photographed and stored in thememory space is described in U.S. Pat. No. 5,138,460 by named inventorsEgawa and Akira entitled “Apparatus for forming Composite Images” issuedon Aug. 11, 1992.

Although these techniques are useful, they are not without theirshortcomings. One shortcoming is the expense and complication arisingfrom the need for orientation and position sensors for determiningpicture orientation. Accordingly, a need exists for a device and methodto create panoramic pictures without the need and expense of positionsensors.

Another shortcoming of the current panoramic cameras is how difficult itis to properly overlap a region between two adjacent frames of apanoramic image. Too much overlap results in wasting memory in thedigital camera with redundant information.

Another shortcoming with current panoramic cameras is their inability toguide the user where the correct edge overlap is required for creating apanoramic picture alignment from two or more images. Accordingly, a needexists to overcome this problem and to guide the user of a camera usinga method and apparatus to overlap two or more images to provide acorrect panoramic picture.

Another shortcoming of the current panoramic cameras is the requirementto overlap regions of two or more images. It is common with panoramicimage generation to stitch together two or more images. Problems at theboundaries of two images include color distortion, perspectivedistortion, pixel adjacency and skipped pixels at the edges when joiningtwo or more images. Accordingly, a need exists for a method andapparatus to overcome these shortcomings.

Another shortcoming of the current panoramic cameras is their inabilityto integrate these mechanisms with other products. The bulk, expense,and complication make these designs difficult to integrate. Accordingly,a need exists for a method and apparatus to allow easy integration ofpanoramic capabilities into other electronic devices.

Another shortcoming is the inability to easily capture two or moreimages to create a panoramic scene without the need for expensivecomputational resources in the camera and without the need for expensiveposition sensors. Accordingly, a need exists for a method and apparatusto provide a camera which can capture two or more images so as to createa panoramic image.

Another shortcoming is the requirement of having two distinct buffers totransform the captured image from one coordinate system to anothercoordinate system, such as rectilinear coordinates to cylindricalcoordinates. The expense and space required for two buffers forcoordinate transformations is undesirable. Accordingly, a need exists toprovide a method and apparatus that can transform images from onecoordinate system to another coordinate system, without the use of twobuffers.

Another shortcoming with current panoramic cameras is the perspective ofthe series of images relative to each other is lost. One known procedureuses sub-sampling. The picture is scaled down so that the horizontaldimension will match the horizontal extent of the view-port. However,the picture must become a narrow stripe in order to maintain the correctaspect ratio. This will produce a great loss of vertical resolution.Another solution is to scroll the panoramic image horizontally by afixed amount. In this case no sub-sampling is used, provided that thevertical dimension of the picture will fit the vertical dimension of theview-port. Accordingly, a need exists to provide a method and apparatusto create a moving display of still pictures with perspective withoutthe need to sub-sample or to horizontal scroll.

Still another shortcoming with current panoramic cameras is therequirement that all the image processing electronic circuitry for thefinal stitching together of one or more images in a series into a singlepanoramic image is integrated into the camera. Many times, the addedexpense, the additional weight and the additional size of the electroniccircuitry make the camera unwieldy to carry and operate. Moreover, theadditional electronic circuitry makes the camera more expensive tomanufacture. Accordingly, a need exists for a method and apparatus toenable the preview of a series of adjacent images to form a panoramic,but to enable the final image stitching processing on a remote devicesuch as a personal computer.

SUMMARY OF THE INVENTION

A method for converting a rectilinear image and a focal length into acylindrical image parameterized by a height of the cylinder and anangular distance in a single buffer on a remote processing device. Themethod, on the remote processing device, comprising joining two or moreimages together to form a panoramic image with corrected perspective.The rectilinear transformation is “in-place” and requires only onebuffer. Color correction and motion estimation is also carried out onthe remote device.

In an alternate embodiment, a computer readable medium corresponding tothe above method is described.

BRIEF DESCRIPTION OF THE DRAWINGS

The subject matter which is regarded as the invention is particularlypointed out and distinctly claimed in the claims at the conclusion ofthe specification. The foregoing and other objects, features, andadvantages of the invention will be apparent from the following detaileddescription taken in conjunction with the accompanying drawings.

FIG. 1 is a block diagram of a digital still camera according to oneaspect of the present invention.

FIG. 2 is a block diagram of the picture stitching device of FIG. 1 inanother embodiment integrated as a standard cell in a semi-customsemiconductor device, according to the present invention.

FIG. 3 is a block diagram of a series of individual horizontal stillimages comprising a panoramic image with overlapping regions, accordingto the present invention.

FIG. 4 is a flow diagram of the overall image acquisition process of thehorizontal images of FIG. 3 using the digital still camera of FIG. 1,according to the present invention.

FIG. 5 is an illustration of a Previous Picture for demonstrating thecomposite overlay and perspective correction of the preview strip theoverall image acquisition process of FIG. 4, according to the presentinvention.

FIG. 6 is an illustration of a selected overlap region of FIG. 5,according to the present invention.

FIG. 7 is an illustration of the corrected perspective of overlap regionof FIG. 6 for aligning the Ideal Next Picture, according to theinvention.

FIG. 8 is an illustration of the Current Picture with an Ideal NextPicture guided by the overlap generated in FIG. 7, according to thepresent invention.

FIG. 9 is a high level block diagram of a picture stitching device ofFIG. 1, according to the present invention.

FIG. 10 is a detailed flow diagram of the stitching overview of FIG. 4,according to the present invention.

FIG. 11 is an illustration of alpha blending mixing between the previewstrip and the Current View, according to the present invention.

FIG. 12 is an illustration of the results of alpha blending of FIG. 11with a constant ratio, according to the present invention.

FIG. 13 is an illustration of the results of alpha blending of FIG. 11with a progressive ratio according to the present invention.

FIG. 14 is an illustration of the results of interlace blending of FIG.11 using blocks, according to the present invention.

FIG. 15 is an illustration of the results of interlace blending of FIG.11 using lines, according to the present invention.

FIG. 16 is an illustration of the results of alpha blending withinterlace blending of blocks of FIG. 15, with the vertical dimensionequal to the vertical dimension of the picture according to the presentinvention.

FIG. 17 is an illustration of the results of alpha blending withinterlace blending of only some of the blocks of FIG. 15, according tothe present invention.

FIG. 18 is an illustration of the areas of measurement for two adjacentfigures, such as images A and images B of FIG. 3, according to thepresent invention.

FIG. 19 is an illustration of the individual frame buffers laid out on apanorama, according to the present invention.

FIG. 20 is an illustration of the panorama of FIG. 19 saved asfixed-sized strips, according to the present invention.

FIG. 21 is a detailed block diagram of the picture stitching device ofFIG. 9, according to the present invention.

FIG. 22 is a block diagram of the Alignment Preview architecture of FIG.21, according to the present invention.

FIG. 23 is an illustration of the capturing of two rectilinear imageplanes and the relationship to a cylindrical circle of a fixed focalpoint f, according to the invention.

FIG. 24 is an illustration of pixels in the overlap region of thecurrent picture of FIG. 23 being projected back to the Previous Pictureof FIG. 23 in the overlap region, according to the present invention.

FIG. 25 is an illustration of pixels of the selected overlap region ofsource picture of FIG. 7 that cannot be perspectively corrected,according to the invention.

FIG. 26 is a block diagram of an alternate embodiment of FIG. 9 with twooptional blocks used to improve the perspective correction algorithmperformance, according to the present invention.

FIG. 27 is an example of a limit area copy from a Previous Buffer to aCurrent Buffer of FIG. 26 during a first pass, according to the presentinvention.

FIG. 28 is an example of a limit area copy from a Previous Buffer to aCurrent Buffer of FIG. 26 during a second pass, according to the presentinvention.

FIG. 29 is an example of a limit area copy from a Previous Buffer to aCurrent Buffer of FIG. 26 during a third pass, according to the presentinvention.

FIG. 30 is an example of a limit area copy from a Previous Buffer to aCurrent Buffer of FIG. 26 during a fourth pass, according to the presentinvention.

FIG. 31 is an example of areas of Current Buffer for which there is noinformation, according to the present invention.

FIG. 32 is an illustration of rectilinear and cylindrical images,according to the present invention.

FIG. 33 is a schematic showing the two phases of the in-placerectilinear to cylindrical transformation process, according to thepresent invention.

FIG. 34 is a block diagram of the Displacement Estimation architectureof FIG. 21 according to the present invention.

FIG. 35 is an illustration of vertical and horizontal downsampling of animage, according to the present invention.

FIG. 36 is an illustration of the sum-of-absolute differences (SAD)between a pair of images according to the present invention.

FIG. 37 is an illustration for determining the optimum displacementcorresponding to the minimum local minima in the profile, according tothe present invention.

FIG. 38 is a block diagram of the Color Correction architecture of FIG.21 according to the present invention.

FIG. 39 is a schematic illustrating the color correction applied atvarious regions of the Previous Buffer and Current Buffer, according tothe present invention.

FIG. 40 is a block diagram of the Stitching and Blending architecture ofFIG. 21, according to the present invention.

FIG. 41 is a schematic illustrating the regions of the Previous andCurrent Buffer, during image compositing, according to the presentinvention.

FIG. 42 is a schematic illustrating the regions of the Previous andCurrent Buffer being composited over a narrower region, according to thepresent invention.

FIG. 43 is a schematic illustrating the regions of.the Previous andCurrent Buffer contributing to the final panorama, according to thepresent invention.

FIG. 44 is a schematic illustrating the Current View within thePanorama, according to the present invention.

FIG. 45 is a high level block diagram of an architecture for a systemusing a motion playback device according to an aspect of the invention.

FIG. 46 is a diagram illustrating a motion picture simulation principle,according to the invention.

FIG. 47 illustrates cylindrical and planar picture representations of apanoramic picture, according to the present invention.

FIG. 48 is a detailed block diagram of the motion playback deviceinternal architecture of FIG. 45, according to the present invention.

FIG. 49 shows an X mapping block architecture of FIG. 49, according tothe present invention.

FIG. 50 shows a Y mapping block architecture of FIG. 49, according tothe present invention.

FIG. 51 is a block diagram of the picture processor of FIG. 49,according to the present invention.

FIG. 52 shows a vertical subsampling architecture, of the pictureprocessor of FIG. 50, according to the present invention.

FIG. 53 shows an embodiment of a cubic interpolation system of FIG. 51,according to the present invention.

FIG. 54 shows an embodiment of a linear interpolation system of FIG. 51,according to the present invention.

DETAILED DESCRIPTION OF AN EMBODIMENT

However, it should be understood that these embodiments are onlyexamples of the many advantageous uses of the innovative teachingsherein. In general, statements made in the specification of the presentapplication do not necessarily limit any of the various claimedinventions. Moreover, some statements may apply to some inventivefeatures but not to others. In general, unless otherwise indicated,singular elements may be in the plural and visa versa with no loss ofgenerality.

1. Glossary of Terms Used in this Disclosure

Actual Next Picture—the picture that is actually captured as the nextaddition to the set of pictures constituting the Panorama.

Bottom-to-Top Vertical Panorama—a Panorama captured by taking a set ofpictures by rotating the camera up (from bottom to top) between eachcapture, with as little horizontal displacement as possible.

Current Buffer—a cylindrically warped version of the current picture.

Current Picture/Current View/Current Frame—a picture displayed on thecamera LCD screen and that is updated in real time. If the LCD screen isnot used or if there is no LCD screen, it is the picture that would becaptured at any given moment if the capture button was pressed.

Ideal Next Picture—in a Left-to-Right Horizontal Panorama, the picturethat would be obtained if the camera was positioned so that the PreviousPicture and Ideal Next Picture have an Overlay Part Length equal to theSet Overlay Part Length and no vertical displacement. In a Left-to-RightHorizontal Panorama, the overlay part is on the right of the PreviousPicture and on the left of the Ideal Next Picture.

Ideal Position of the Camera for the Ideal Next Picture—The position ofthe camera that allows to capture the Ideal Next Picture.

Image Stitching—the process of digitally combining scenes taken fromdifferent camera orientations.

Left-to-Right Horizontal Panorama—a Panorama captured by taking a set ofpictures by rotating the camera clockwise (from left to right) betweeneach capture, with as little vertical displacement as possible.

Overlay Part Length—in a Horizontal Panorama, the width of the overlaypart. It is expressed in term of percentage of the whole picture width.

Overlay Part of A Picture—the part of the picture picturing the overlayzone.

Overlay Zone of Two Pictures—the part of a scene that is present in thetwo pictures.

Panoramic—an image with at least one dimension such as height or widthwhich is greater in dimension of a single capturing device and ofteninvolves a series of images. A picture created from a set of picturesand that has at least one dimension bigger than the correspondingdimensions of a source picture.

Preview Strip—a computed image created through digital processing of theoverlay part of the Previous Picture and that strives to predict whatthe overlay part of the Ideal Next Picture will look like.

Previous Buffer—a cylindrically warped version of the Previous Picture.

Previous Picture/Previous Frame—a picture that has already been capturedand that is the latest addition to the set of picture constituting thePanorama.

Right-to-Left Horizontal Panorama—a Panorama captured by taking a set ofpictures by rotating the camera anticlockwise (from right to left)between each capture, with as little vertical displacement as possible.

Set Overlay Part Length—a constant Overlay Part Length for each pair ofpictures constituting the Panorama. The Set Overlay Part Length is thefixed length chosen for a given Panorama.

Top-to-Bottom Vertical Panorama—a Panorama captured by taking a set ofpictures by rotating the camera down (from top to bottom) between eachcapture, with as little horizontal displacement as possible.

2. Picture Stitching Camera

Referring now in more detail to the drawings in which like numeralsrefer to like parts throughout several views, shown in FIG. 1 is a blockdiagram of a digital still camera 100 according to one aspect of thepresent invention. A digital camera comprises one or more optical lenses102 with an autofocus/shutter, driver and actuator 104 and associatedphotometry interface 108 such as autofocus, auto shutter and contrastcontrol. An imager 106 such as an CCD or equivalent 104 converts animage projected through optical lens 102 to a series of pixels 106.Regulators and image drivers 108 allow regulation of the imager 106. Anaudio acquisition device 212, such as microphone, along with audioprocessing circuitry 214, allows a user to make aural recordings alongwith digital images. A battery 118 with power management circuitry 118allows the camera 100 to work as a portable device. A picture processor116, provides pixel information to one or more frame buffers 118 coupledto picture stitching device 120 which is described further below. Inthis embodiment, the picture stitching device 120 is implemented as anASIC. A LCD display 122 or equivalent enables a user to view the imageprojected through lens 102 into imager 106 and controlled by LCDcontroller 134. A PAL/NTSC 124 encoder provides an interface to otherdisplay types. An image CODEC 132 coupled to picture processor 116provides known image enhancement effects for the picture processor 116.A DSP 124 such a STMicroelectronics ST-20/SH3-DSP is used to control thememory interface and the data I/O 126 such at Infra-Red, UniversalSerial Bus or other interfaces. A DRAM provides execution memory for theDSP 124 to perform Image Stitching algorithms as described below. Anaudio output 130 such as a speaker provides the user aurally playback.All of these components are representative components of the digitalcamera 100. Storage media 136 such as Flash memory, diskettes orremovable hard drives store each image and associated audio. In oneembodiment, the algorithms carrying out the steps for Image Stitchingdescribed below are stored on the storage media 136 along with capturedimages and audio. Processing for the images may occur prior to or afterthe image is store in storage media 136. The general operation of adigital camera comprising most elements described herein is wellunderstood and those skilled in the art.

One or more user inputs via the LCD Controller 128 provides user controlover camera functions such as the orientation of the panoramic e.g.,horizontal or vertical, and the direction of movement such as aLeft-to-Right Horizontal Panoramic, a Right-to-Left HorizontalPanoramic; a Top-to-Bottom Vertical Panoramic; and a Bottom-to-TopVertical Panoramic. Other user input such as the optional features anddesired effects and to set system parameters such as:

Panorama mode on/off.

Panorama parameters.

Left-to-Right Horizontal mode.

Right-to-Left Horizontal mode.

Top-to-Bottom Vertical mode.

Bottom-to-Top Vertical mode.

Set Overlay Part Length.

Preview display on/off.

Mixing mode parameters.

Alpha blending on/off.

Alpha blending parameters such as alpha blending ratio.

Interlaced block mode on/off.

Interlaced block pattern selection.

In another embodiment, many of the components of FIG. 1 are embedded inas part of a standard cell in a semi-custom semiconductor device. FIG. 3is an block diagram 300 of the picture stitching device of FIG. 2 inanother embodiment integrated as a standard cell in a semi-customsemiconductor device, according to the present invention. In thisembodiment, the picture stitching device 124 and picture processor 120are physically implemented as one unit. The DSP 134, serial I/O 136,image CODEC 132, LCD preview and display 126, PAL/NTSC encoder 130 andLCD controller & drivers 126 are all integrate as one device 200. Tothose skilled in the art, it will be obvious to substitute and modifythis exemplary single device 300 to include the picture stitching device124 and DSP 224 with other circuitry as well.

3. Panoramic Capturing Overview

Turning now to FIG. 3 shown is a series of four images 300 labeled A, B,C, and D with overlap edges parts ab, bc, cd between. These four imagesare joined together to form one Panoramic image. It is important to notethat the Panoramic images can be from Left-to-Right as shown in FIG. 3or Right-to-Left for a Horizontal Panoramic. For a Vertical Panoramic,(not shown) the series of images can run from top- to-bottom or frombottom-to-top. Accordingly, this present invention is not limited to onetype of horizontal or vertical orientation, and four types of panoramicimages are disclosed: a Left-To-Right Horizontal Panoramic; aRight-to-Left Horizontal Panoramic; a Top-to Bottom Vertical Panoramic;and a Bottom-to-Top Vertical Panoramic.

FIG. 4 is a flow diagram 400 of the overall image acquisition using thedigital still camera of FIG. 1, according to the present invention.After this overall process is described, a more detailed description ofthe process steps of FIG. 4 are described below. The digital camera 100acquires a series of images to form a panoramic image, such as theseries of Left-To-Right Horizontal Panoramic A, B, C, D with overlapedge parts ab, bc, cd of FIG. 3. The overall image acquisition processbegins with the initialization of the digital camera 100. The userinputs from the LCD controller setting up the orientation of thepanoramic, such as horizontal or vertical and other system parametersfor picture stitching device 124 are set, step 402. The next image islocated in the view finder of digital camera 100. Returning to theexample in FIG. 3, if the image is A, it is the first image, if theimage is B,C, or D it is the next image and the regions of overlap ab,bd, and cd for each image pair is made as a composite. Once the imageeither the first frame or the successive frames with the desired overlapare aligned in the view finder of digital camera 100, the so calledIdeal Next Picture, is created. The region of overlap, called thePreview Strip, is a composite of both the Previous Picture and theCurrent Picture. The Preview Strip comprises a perspectively correctedPrevious Picture in the region overlap. The Preview Strip assists theuser in aligning the Current Picture and the Previous Picture. TheCurrent View is stitched together with the Previous View, in the stitchframe step, 408. A test is made to see if the last image had beenstitched. Referring again to FIG. 3 that is whether image D has beenstitched with image C using overlap region cd. If the last image is notbeen stitched then the process returns to step 404 until the last imagehas been stitched, step 410. Once the last image is stitched finalcorrections to the overall image and overall data is made, step 412.After the first frame is acquired, a perspectively-corrected strip ofthe Previous Frame is overlaid on the Current Frame to aid the user inacquiring the Current Frame. For example, if a Horizontal Panorama isbeing acquired from left-to-right, a strip from the right portion of thePrevious Frame is perspectively corrected and overlaid on the leftportion of the Current Frame in the viewfinder. After each frame isacquired, the stitch module is invoked with the new frame buffer. Thestitch module aligns and blends the new frame into the Panorama, andincrementally saves out the Panorama. After the last frame is acquiredand stitched, the Finalize Stitch module is called to save out anyremaining data.

Turning now to FIGS. 5-8 is an illustration of a series of Left-to-RightHorizontal Panorama images illustrating the perspective correction ofFIG. 4 as seen by the user of the digital camera 100, according to thepresent invention. Note the capturing of the Current Picture with anIdeal Next Picture guided by the overlap generated edge 602. In FIG. 5,an image is captured in the digital camera 100, this becomes thePrevious Picture 500. A Preview Area 602 is shown from the PreviousPicture 502 to user in FIG. 6. FIG. 7 illustrates how the Preview Area602 is perspectively corrected or warped 702 to assist the user inaligning the Current Picture 800 in FIG. 8. Notice the perspectivelycorrected Preview Area 702 makes aligning the Current Picture 800 mucheasier. As shown in FIG. 8, a Preview Strip 702 is generated from theoverlay part 602 of the Previous Picture 500. Note the similaritybetween the generated Preview Strip 702 and the Current View 800 overlaypart. The computed Preview strip 702 is used as a visual clue, displayedon the LCD 126 of digital camera 100, to help the user position thecamera 100 until the Current View 800 matches the Preview Strip 702.

The computed Preview Strip 702 can be displayed on the digital camera100 in various way to help the user in precisely placing the camera inthe desired position. One way is to alpha blend (with adjustablecoefficient) the preview with the current display of the scene on theLCD 126. In another embodiment, the interlace some blocks of the previewwith the displayed Current View on the digital camera 100. Furtherinformation on blending is described below.

4. High Level Picture Stitching Device Architecture

FIG. 9 is a high level block diagram 900 of a picture stitching device124 of FIG. 1, according to the present invention. The Frame Buffers 122comprise a Previous Buffer 902 (or prevBuf) and a Current Buffer 904 (orcurBuf). The Previous Buffer 902 and Current Buffer 904 are coupled tothe picture stitching device 124 through a set of address 906 and datalines 908. A clock input 912 provides timing inputs to the picturestitching device 124. A picture input from the imager 106, provides thedigital image to the picture stitching device 124. An output labeled outpicture 916, provides an output from the picture stitching device 124 toother devices such as the Image CODEC 132 and DSP 134. An input labeledControl settings 910, provides parameter input to picture stitchingdevice 124. Parameter input includes, the size of the Previous Buffer902 and the Current Buffer 904, the amount of overlap region betweenimages, types of color correction, and more. The Previous Buffer 902 isa memory devices containing the Previous Picture, its sub-sampledversion and all the information related to it and required by thealgorithms. The Current Buffer 904 is memory devices containing theCurrent Picture, its sub-sampled version and all the information relatedto it and required by the algorithms. The two buffers can be implementedeither as two separated physical devices or one physical device and inthis case the distinction is logical only. The stitching device requiresfour types of buffers of different sizes: (1) two working buffers eachthe size of the image frame of the digital camera; (2) one buffer forthe overlay area or Preview Strip; (3) four down sample working buffersfor motion estimation, described later on; and (4) several other smallerbuffers and registers for calculations. The detailed process flow ofstitching using picture stitching device 124 is now described.

A. Stitching Overview

The stitching overview of FIG. 4 is now described in further detail.FIG. 10 is a detailed flow diagram of the stitching overview of FIG. 4,according to the present invention. Each of these process steps aredescribed further below along with an exemplary hardwareimplementations. To aid with the understanding of the concepts in theprocess flow of FIG. 10 for the Previous Buffer 902 and Current Buffer904 references are made to FIG. 18 which illustrates the areas ofmeasurement for two adjacent images of FIG. 3, such as images B and C.The stitching process is managed around the two working frame buffers122 the Previous Buffer 902 and the Current Buffer 904. In general thePrevious Buffer 902 contains a cylindrically warped version of thePrevious Picture and the Current Buffer 904 contains a cylindricallywarped version of the Current Picture 800. To illustrate the digitalimage contents of the Previous Buffer 802, the term prevBuffer 1802 isused. The prevBuffer 1802 has a preFarEdge 1804 which corresponds to theleft edge of the Previous Buffer 902 and a preNearEdge 606 whichcorresponds to a right edge of the Previous Buffer 902. The curBuffer1808 is a graphically illustration of the digital image contents storedin Current Buffer 904. The curBuffer 1808 has curFarEdge 1810 whichcorresponds to the left edge of the Current Buffer 904 and curNearEdge1812 which corresponds to the right edge of the Current Buffer 904. Ingeneral the near and far edges do not have to correspond to the trueedges of the Previous Buffer 902 or the Current Buffer 904. In avertical panoramic orientation, the curFarEdge 1810 and the curNearEdge1812 correspond to the top edge of the Current Buffer 904 and the bottomedge of the Current Buffer 904 respectively. The Previous Buffer 902also has the top edge and bottom edge correspond to the prevFarEdge 1804and the prevNearEdge 1806. The overlap region 1818 is defined by thecurFarEdge 1810 on the left and the preNearEdge 1806 on the right.

The process flow of stitching is divided into two cases: (1) stitchingthe first frame, and (2) stitching the subsequent frames. The processflow begins, step 1002 and a determination if the first frame has beencaptured, step 1004. If the frame is the first frame, such as A of FIG.3, the Current View is loaded directly into the prevBuffer 1802, step606. In step 1008, the desire amount of overlay between successiveimages is computed, such as the overlay regions ab, bc and cd of FIG. 3.In one embodiment, the amount of overlay is a system parameter that isset by the user. An in place transformation such as a rectilinear inplace transformation to perspectively correct the Overlay Buffer 1818(ovrlyBuffer).

The overlay region 1818, also referred to as the Preview Strip 702,contains a picture obtained from the previous picture and which andwhich has been transformed digitally in order to match the perspectivefor the ideal current picture. This overlay picture contained in theoverlay region 1818 is used to guide the user in position in the cameraso that he/she will capture a current picture as close as possible tothe ideal current picture. The image contained in the overlay region1818 can be obtained from the previous picture in cylindricalrepresentation or from the previous picture in rectangularrepresentation. Obtaining the overlay picture from the cylindricalrepresentation does not require any additional processing block, for itis the same transformation used in the Motion Picture Playback Moduledescribed further in Section 6 “Motion Play-Back of Still Picture ThatHave Been Previously Save”, below. However, in the this embodiment, theoverlay region 1818 is in a rectangular representation. The use of arectangular representation yields a better picture quality since onlyone digital transform is used while two are needed when using thecylindrical source. The use of an in-place cylindrical transform, meansthat the rectangular coordinate picture is overwritten by thecylindrical coordinate picture. And it is necessary to perform thecomputation and saving of the overlay region 1818 in the overlay bufferprior to or before the in-place rectilinear-to-cylindrical transformoccurs. Once the overlay region 1818 is estimated to correct perspectivefor the next image is calculated, step 610. The parameters for stitchingare set as follows:

preFarEdge=0;

preNearEdge=length;

prevDispl=default.

Where length is the length of overlap between the images, and default isa displacement value whose value is set to be greater than zero and lessthe length of overlap. The value is the displacement along normal to thedirection of orientation for the panoramic. Stated differently, for ahorizontal panoramic, the default would be the maximum verticaldisplacement. And the stitching for this first frame is complete, step1016. The Current View is loaded into the curBuffer 1808, step 1019, apreview of the curBuffer 1808 into the overlay region 1818 of theCurrent Buffer 904 is done to assist the user in aligning the two imagesat the region of overlap. Returning to FIG. 3, the region of overlapbetween the first image A and the second image B is ab. The user istrying to capture the Ideal Next Picture by positioning the digitalcamera 100 to align the region of overlap using a perspectivelycorrected Preview Strip 702. As described below, the Previous Picture500 and the Current Picture 800 at the region of overlap aresuperimposed, one on top of the other to assist the user in the IdealPosition of the camera 100. Once the image is captured, it iscylindrically warped in place, step 1022. At this point both theprevBuffer 1802 and the curBuffer 1808 are in cylindrical coordinates.Next, the displacement between the previous and current picture (alsocalled motion estimation), between the prevBuffer 1802 and the curBuffer1808 is computed along with color correction between the prevBuffer 1802and the curBuffer 1808, step 524. In one embodiment, the colorcorrection is performed on the prevBuffer 1802 between prevFarEdge 1804and preNearEdge 1806. And at the same time, color correction is alsoperformed on curBuffer 1808 from curFarEdge 1810 up to the curNearEdge1812. Color correction is not performed on the region of prevBuffer 1802that has already been saved out, step 1014. After the color correctionthe overlap region of the two buffers are linearly composited, steps1026 and 1028. Note in another embodiment, the color correction isperformed in combination with areas of the picture already save out, toyield a smoother color variations since the color change could graduallyspeared on a bigger surface.

Next, the prevBuffer 1802 from prevFarEdge 1804 to preNearEdge 1806 withpreDisp (previous displacement or previous motion estimation ) is saved,step 1030. The region of prevBuffer 1802 between prevFarEdge 1804 andprevNearEdge 1806 is save out to image memory, such as storage media142. Since all the prevBuffer 1802 would have been saved out a thistime, the contents of Previous Buffer 1602, that is the image prevBuffer1802 are no longer needed. Now the curBuffer 1808 becomes the prevBuffer1802 and the prevBuffer 1802 becomes the curBuffer 1808. Stateddifferently the curBuffer 1808 and prevBuffer 1802 are swapped. Finally,the system parameters are set as follows:

prevFarEdge=length−curFarEdge;

prevNearEdge=length;

prevDisp=curDisp.

Now further details on the Picture Stitching Device 124 are described.The description begins with an exemplary hardware implementationfollowed by examples of the processes implemented by each hardwaresection.

B. Mixing the Preview Strip With the Current View

There are several methods to mix the preview strip with the CurrentView. Methods that are discussed are: Alpha Blending; Alpha Blendingwith Progressive Ratio; Interlacing; Alpha Blending & Interlacing.

The alpha blending mixing between the preview strip and the Current Viewis illustrated in FIG. 11. The preview strip is alpha-blended with theoverlap part of the real-time Current View and displayed. Thealpha-blending operation is done with the following algorithm:

 Result=alpha×Preview_Strip+(1−alpha)OverlapPart_of_Current_View.

The alpha ratio of the blending can be user selected of fixed. The alphablending can be uniform over the preview picture or varies over theoverlay part as shown in FIG. 11, where the Current View is denoted inblack 1102 and the preview strip denoted in white 1104. The result ofthe constant ration alpha blending in shown in FIG. 12. FIG. 13 is anillustration of the results of alpha blending of FIG. 11 with aprogressive ratio according to the present invention. An error inpositioning the camera will be immediately noticed on the display as adoubling feature of the viewed scene.

Blocks of the preview picture are interlaced with the real-time CurrentView displayed on the camera. Several block sizes can be used, includingblocks as wide as the preview picture (blocks of lines) and blocks ashigh as the preview picture (blocks of columns). The size of the blocksdoes not need to be uniform. FIG. 14 is an illustration of the resultsof interlace blending of FIG. 11 using blocks and FIG. 15 is anillustration of the results of interlace blending of FIG. 11 using linesaccording to the present invention. An error in positioning the camerawill be immediately noticed on the Display as a breaking of features ofthe viewed scene at the blocks edges.

Both alpha Blending and Interlacing can be used as shown FIG. 16, whichillustrates the results of alpha blending with interlace blending ofblocks of FIG. 15 and FIG. 17 which illustrates the results of alphablending with interlace blending of only some of the blocks of FIG. 15according to the present invention. Alpha blending on blocks areillustrated with the vertical dimension equal to the vertical dimensionof the picture.

C. Saving the Panorama

FIG. 19 shows the individual frame buffers laid out on a Panorama. Thewidth of the Panorama grows as more frames are acquired. In oneembodiment, the height of the Panorama is fixed, and is defined as afunction of the height of each individual frame and the maximum verticaldisplacement. A fixed vertical height enables the calculation of anindividual frame, combined with the previous vertical displacement ofthe previous frames so that if the frame extends beyond the fixed heightof the panorama is clipped. In another embodiment, the height of thepanorama can be adjusted when the vertical displacement of an individualframe combined with eh vertical displacement of all previous frames issuch that the frame extend beyond the current height of the panorama. Ifthe vertical displacement of an individual frame is such that the frameextends beyond the height of the Panorama, the frame is clipped.

The Panorama is saved in fixed size strips as illustrated in FIG. 20.The use of saving the image as strips helps facilitate the playback ofthe strip, as described in Section 6 “Motion Play-Back of Still PicturesThat Have Been Previously Stored,” below. The saving process is managedby a Panorama Saving Module that maintains a working buffer the size ofa single strip. Each time a region of prevBuffer 1802 is saved, as manystrips as necessary are saved out, and the remainder is stored in theworking buffer (to be saved out the next time). After the last curBuffer1808 is saved, the (partial) working buffer is saved out.

5. Detailed Picture Stitching Device Architecture

FIG. 21 is a detailed block diagram 2100 of the picture stitching device124 of FIG. 19, according to the present invention. The I/O Sections2102 contains all the Interfaces and data formatting resources able tomake the device communicate with the environment. Two lines 2134 and2136 from Previous Buffer 1902 and Current Buffer 1904 are coupled tothe Alignment Preview 2104, Displacement Estimation 2106, ColorCorrelation Parameters Estimation, Stitching & Blending 2110 and thePicture-through-Picture 2120. The output of the Alignment Preview isfed-back to the I/O Section 2102. The output of the DisplacementEstimator 2106 and the Color Correction Estimator 2108 are feed to aRegister 2112, which feeds the result into the final Stitching andBlending 2110. The video Mode & Geometric Mapping 2116 receives picturedate from I/O Section 2102. The output of the Stitching & Blending 2110and the Picture-through-Picture 2120 and the Video Mode & GeometricMappings is feed through Selector 2114 which is controlled by FunctionSelection and Control 2118. The function selection and control 2118include on/off controls 2138 to each major section as shown. Thearchitectures of Alignment Preview 2104, Displacement Estimation 2106,Color Correlation Parameter Estimator 2108, and Stitching and Blending2110 are now described followed by the process implemented by each.

In an alternate embodiment, everything except the minimum circuitry ofthe picture stitching device 124 to enable alignment preview 2104 can belocated on a remote processing device such as a personal computer orother microprocessor-based device. The use of a remote processingdevice, reduces the size of the electronic circuitry in picturestitching device 124, the weight of the picture stitching device 124 andthe complexity and associated cost of manufacturing the picturestitching device 124. In this embodiment, any or all of the followingcomponents may be in a remote processing system, including: thedisplacement estimation 2106, the color correction parameter estimation2108, and the stitching & blending 2110 can be located or implemented ina remote microprocessor-based system (not shown). The interface to theremote microprocessor-based system may be through the Serial I/O 136 orStorage Media 142 that may be removable and coupled to the remotemicroprocessor-based system.

A.1. Alignment Preview Architecture

FIG. 22 is a block diagram 2200 of the architecture of the AlignmentPreview of FIG. 21, according to the present invention. After thePrevious Picture and Current Picture have been converted intocylindrical coordinates, the relative displacement between them must becomputed.

A.2. Perspective Correction in Alignment Preview

An overview of the picture stitching device 124 is now described for theregion of overlap. FIG. 23 is a top view of the Alignment Previewgeometry according to the present invention. A Previous Picture 2302 wascaptured by digital camera 100 and that f is the focal length of thelens 102 of this camera. This Previous Picture 2302 is stored in thePrevious Buffer 902. The digital camera 100 is rotated by an angle δθ sothat the Current Picture 2304 is captured with a given overlap regionalong the x axis with the Previous Picture 2304. The symbol δθ denotesthe rotation angle required to produce an overlap of the given width.The symbol x_ovlp 2310 denotes the right edge of the overlay region;x_ctr 2308 denotes the center of the image plane. It is important tonote, that in this embodiment, the rotation is horizontal, fromleft-to-right but the principle applies described herein apply tohorizontal right-to-left rotation as well as top-to-bottom orbottom-to-top rotation.

The part of each picture (Previous Picture 2302 and Current Picture 2304that is also depicted in the other picture is called the overlap region.Although the scene captured in the Previous Picture 2302 and the CurrentPicture 2304 is the same in the overlap region 2306 because of adifference in perspective. The Alignment Preview 2304, generates apreview of the Current Picture 2304 by correcting the perspective of theoverlap region 2306 of the Previous Picture 2302 to conform with theperspective of the Current Picture 2304.

The process of perspective correction during the alignment previousinvolves the following steps:

(a) Mapping each of the pixels of the Previous Picture 2302 to an X-Ycoordinate;

(b) Mapping each of the pixels of the Current Picture 2304 to an X-Ycoordinate so that the x-axis is the center of the image plane of thecurrent picture along the direction of the panning motion of the digitalcamera 100 and so that the y-axis is perpendicular to the x-axis andlies in the same image plane. This is shown graphically in FIG. 23.

(c) Calculating the rotation angle δθ based on the amount of overlap inthe preview area along the x direction.

(d) Calculating the rectilinear x-coordinate transformation based on δθto project or perspectively warp, the Current Picture 2304 onto PreviousPicture 2302.

This is shown graphically in FIG. 24.

(e) Calculating the y-scanline resealing.

Steps c, d, and e above are all now shown mathematically with referenceto FIGS. 24 and 25.

FIG. 24 is an illustration 1500 of pixels in the overlap region of thecurrent picture of FIG. 23 being projected back to the Previous Pictureof FIG. 23 in the overlap region 2306 according to the presentinvention. We note x_dst and y_dst the coordinate of the pixels of theCurrent Picture 2304 and we note x_src′ and y_src′ the coordinate of thepixels of the Previous Picture 2302.

In this example, the overlap region 2306 of the Current Picture 2304indicates the overlay region 2306 that is derived from the PreviousFrame 2304; in this case, the camera 100 rotating clockwise.

To calculate the rotation angle δθ based on x_ovlp 2310:

δθ=arctan(x _(—) ctr−x _(—) ovlp/f)+arctan(width−x _(—) ctr/f).

The width of this overlap region 2306 is a system parameter. Given thewidth of the overlap region 2306, the rotation angle required to producean overlap of that width is calculated. The width of the overlap regionis specified by the location of the region's rightmost (innermost) edgex_ovlp 2310.

The transform induced on the Previous Picture as a result of the 3Drotation is a uniform resealing of each vertical scanline independently.(In the case of camera tilt, the transform is a uniform resealing ofeach horizontal scanline independently). This transform is described by:(a) the location of the source scanline corresponding to eachdestination scanline; (b) the scale factor for each scanline; and (c)column scaling. Calculating the x-coordinate transformation:

 x′ _(—) src=(x _(—) dst×cos δθ)+(f×sin δθ), f′ _(—) src=(−x _(—)dst×sin δθ+(f×cos δθ).

Solving for x-src:

x _(—) src=(f×x′ _(—) src)/f′ _(—) src x _(—) src=f×((x _(—) dst×cosδθ)+(f×sin δθ))/((−x _(—) dst×sin δθ)+(f×cos δθ))  EQ1:

The location of the source scanline (x_src) is dependent on the rotationangle (δθ) the focal length (f), and the destination scanline location(x_dst). The equations above describe a 2D plane rotation of the x and faxis; the y axis is left unchanged. The source scanline location (x_src)is calculated by projecting onto the image plane of the Previous Frame.Calculating y-scanline resealing:

y′ _(—) src=y _(—) dst. y _(—) src=(f×y′ _(—) src)/f′ _(—) src y _(—)src=y _(—) dst×[f/((−x _(—) dst×sin δθ)+(f×cos δθ)]  EQ2:

This equation shows that y_src is a uniform resealing of y by a factorthat depends on x_dst, the rotation angle δθ, and the focal length f.Like x_src, y_src is computed by projecting onto the image plane of thePrevious Frame.

Note that there are likely to be instances where the overlap region 2306of Current Picture 2304 cannot be computed from overlap region 2306 ofthe Previous Picture 2302, according to the present invention. This isshown as areas 2502 and 2504 of FIG. 25.

It should be understood that all trigonometric calculations can uselook-up tables as required for faster implementation. All sections ofthe picture stitching device 124 use a ‘closest’ coordinate approachwhen the result of a coordinate computation is not integers. Anotherapproach is to read the value of the neighboring pixels and perform aninterpolation. The location of the source scanline (x_src) is calculatedbased on the rotation angle (δθ), the focal length (f), and thedestination scanline location (x_dst). The principle is that, for anygiven column x_dst of the Preview Area 602, the location in the PreviousBuffer 902 is computed based on the corresponding x_src column. Thegiven column x_dst the scaling factor:

f/((−x _(—) dst×sin δθ)+(f×cos δθ))

The column scaling is performed by reading a column from the PreviousBuffer 902 and write a scaled version of this column in the CurrentBuffer 904.

A.3. Fast Buffering During Perspective Correction

Turning now to FIG. 26, shown is block diagram 2600 of an alternateembodiment of FIG. 9 with two optional blocks 2602 and 2604 used toimprove the perspective correction algorithm performance, according tothe present invention. A buffer copy section copies a block of memoryfrom the Previous Buffer 902 to the Previous Access Buffer 2602. For agiven Previous Fast Access Buffer size, the size of the Previous Buffer902 part than can be buffered depends on the rotation angle (δθ), whichdepends on the desired size of the overlap area 2306 and of the focallength f. FIGS. 27-30 illustrate an example where the complete PreviousBuffer 902 and Current Buffer 904 are copied in four passes. Thecoordinate of the area to be buffered in the Current Buffer 904, thecoordinate of the corresponding points in the Previous Buffer 902 (givenby equations EQ. 1 and EQ. 2 above) and the limits of the area to bebuffered in the Previous Buffer 902 are represented.

FIGS. 27-31 illustrates an example of a limit copy area 2702 from aPrevious Buffer 902 to a Current Buffer 904 of FIG. 26 during fourpasses according to the present invention. In FIG. 28 which illustratespass two, note that the areas to be buffered in the Previous Buffer 902overlap between two successive passes. Also note that some points of theCurrent Buffer 904 have no corresponding points.in the Previous Buffer902 as they fall outside the picture area. This results in area of theCurrent Buffer 904 for which there is no information. This isillustrated in FIG. 31 as area 3102.

B.1. Algorithm for Rectilinear-to-Cylindrical Transform (CylindricalWARP in Place)

FIG. 32 is an illustration of rectilinear and cylindrical images. Arectilinear image is parameterized by its optical center (x_ctr, y_ctr)and its focal length (f). The coordinates of a cylindrical image are theheight of the cylinder (h) and the angular distance (e). The Rectilinearto Cylindrical Transform transforms a rectilinear image (an imageacquired with optics that can be approximated by a model pin-holecamera) into cylindrical coordinates. The specific algorithm outlinedbelow performs the transformation in-place with the use of a smallbuffer. It also automatically crops the result as shown in the righthalf of FIG. 32. Like the Overlay Generator, the Rectilinear toCylindrical Transform uniformly rescales each vertical scanline in theimage independently. Thus, the transform is described by: (a) thelocation of the source scanline corresponding to each destination scanline, and (b) the scale factor for each scanline.

FIG. 33 is an illustration of the two phases of the in-place rectilinearto cylindrical transformation process. The left-to-right phase ends withthe last column being the column just before the first columncorresponding to Δcol<0; the right-to-left phase ends with that column.The in-place implementation of the Rectilinear to CylindricalTransformation comprises two passes (which could be implemented inparallel). The first pass is the left-to-right pass which begins withthe leftmost column and ends just before the first column correspondingto Δcol<0. Before proceeding with either passes, the vertical scanlinecorresponding to the last column of the left-to-right pass is buffered.

During the left-to-right pass, each new destination vertical scanline iscomputed by resealing a scanline somewhere to the right of it (or evenat the same location). Since the image representation of the currentimplementation could be subsampled horizontally, a buffer of two columnof pixels, the current scanline and the next (right), is needed. Whenusing a YcrCb format for the pictures were Chroma samples (Cr: Chromared and Cb: Chroma blue) are decimated. This is because the originalsubsampled chroma samples could be required after they have beenmodified (rescaled). Each Chroma sample is shared between two successiveLuma (Y) samples. At the end of each iteration of the pass, the buffersare updated. If the source vertical scanline is either the current orthe next scanline, the buffered copies are used instead of being derivedfrom the Current Picture.

In another embodiment, interpolation is used to create a column ofsource pixel (in the case when the computed source column coordinatedoes not correspond exactly to a real source column coordinate, theclosest real source column is used. The use of the closest real sourcecolumn is an approximate solution that yields average picture quality.To perform interpolation, several source columns are used so a fewcolumns are buffered for later interpolation.

The right-to-left pass proceeds analogously, the only difference is thatthe source scanlines are computed by resealing scanlines somewhere tothe left of current scanlines. A similar two-scanline buffer thatcontains the current and the next (left) is also required. The last(leftmost) scanline could require the scanline to the left of it. If so,the copy buffered at the start should be used in place of the CurrentPicture.

B.2. Calculating the X-coordinate Transformation

The location of each source vertical scanline (x_src) depends on thedestination scanline (θ) and the focal length (f). The coordinate θvaries linearly from θ_min to θ_max as shown in the following equations.

Where

x_src=the coordinate of a column in the source picture

θ=the Angle coordinate of a column in the transformed picture

The transformation relation:

x _(—) src=f×tan(θ)

where f is the focal length used to capture the source picture.

x_src varies from −w/2 to w/2, and is equal to 0 at the center of thepicture. θ varies from θ_min to θ_max and is equal to 0 at the center ofthe picture. Note that θ_max=f×tan (w/2)=−θ_min.

For storage purpose, a column equivalent is defined to the anglecoordinate and define x_dst is defined such that:

x _(—) dst=θ×f×tan(θ_max)/θ_max

This definition of x_dst ensure the following desirable property:

for θ=0, x_dst=0;

for θ=θ_max, x_dst=f×tan(θ_max)=w/2

for θ=θ_min, x_dst=f×tan(θ_min)=−w/2

i.e. x_dst is centered on the center of the destination scan line andvaries from −w/2 to w/2, just like x_src.

Using this definition it follows that:

x _(—) src=f×tan(θ)=f×tan((x _(—) dst/f)×(θ_max/tan(θ_max)))

to simplify, this can be written

x_src=f×tan(θ)×x_dst), where f and C are constants

By studying the function x_src(x_dst), it is follows that:

for any x_dst between 0 and w/2, x_dst>=x_src(x_dst)>=0

for any x_dst between −w/2 and 0, 0>=x_src(x_dst)>=x_dst

Returning to the in place transformation, in order to avoid ‘gaps’ oroverwriting in the destination image, for each column (i.e. for eachx_dst) of the destination scanline, we find the closest correspondingcolumn (i.e. x_src) in the source image.

It has been show that:

for any x_dst between 0 and w/2,

x_dst>=x_src(x_dst)>=0

This means that, for every x_dst, the information required is located inthe source image at column coordinate x_src, which is smaller or equalto x_dst, never bigger than x_dst. In turn, this means that, for anygiven x_dst, the area of the SOURCE image situated for column coordinatebigger than x_dst is not needed.

By scanning the destination image between 0 and w/2 in decreasing order(i.e. starting at w/2 and finishing at 0).

at time t1, compute x_dst1 using x_src1<=x_dst1

at time t2>t1, compute x_dst2 using x_src2<=x_dst2

and x_dst2<x dst1 since the scan is progressing in decreasing order.

Therefore, it follows:

x _(—) src 2<x _(—) dst 1

All the information of the source image situated at column coordinatebigger than x_dst, is deleted, since it will not be needed for thecreation of the current x_dst column of the destination image nor forthe creation of any column later in the processing.

Given proper care, the discarded area of the source image can be used tostore the destination image itself in order to save the cost of aseparate buffer.

When doing the overwriting of the source image by the destination image,some special care must be taken, in particular:

since the function x_src(x_dst) will most of the time not return aninteger value, other methods may be used to find the closest sourcecolumn matching x_src. For example, interpolating a source column basedon adjacent source columns, a small number of source columns aroundx_dst should be buffered and not overwritten as long as they can benecessary in the computation.

when using a Y, Cr, Cb format (in which Cr and Cb samples are decimated,i.e.: shared between several Y sample columns) for the source image,proper buffering of Y columns must be done around x_dst in order not toloose relevant information.

In the region between −w/2 and 0, the same reasoning apply if, between−w/2 and 0, the destination image is scanned in increasing order (i.e.starting at −w/2 and finishing at 0). So, to perform the in placecylindrical transformation, it is necessary to split the pictureprocessing in two part, one from −w/2 to 0 and the other one from w/2 to0. Again, some special care (buffering of a few columns around x_dst)must be taken around x_dst=0.

Although only a small number of columns must be buffered and the size ofthese buffers is small compared to the size of a complete image buffer.This means that using in place transform save an appreciable amount ofmemory, and the benefits are proportional to the size of the imagesused.

B.3. Calculating Y-scanline Rescaling

y _(—) src=y_scale×y _(—) dst

y′_scale=f/sqrt(tan²θ+1)

y′_scale_min=min(f/sqrt(tan²θ_min+1), f/sqrt(tan²θ_max+1))

y_scale=y′_scanline/y′_scale_min

This equation shows that the destination vertical scanline is a uniformrescaling of the source scanline whose scale factor depends on the focallength (f) and the horizontal coordinate (θ). The scale factor isnormalized such that the scale factor corresponds to unity at either theleft (θ_min) or the right edge (θ_max).

C.1. Algorithm for Displacement Estimation (Motion Estimation)

Motion estimation is performed on a pair of images in cylindricalcoordinates. Motion estimation is carried out in four steps:

(1) Horizontal and Vertical Downsampling;

(2) One-Dimensional High Pass Filtering;

(3) Computation of Displacement of Dominant Direction; and

(4) Computation of Displacement in Other Direction.

During the first step, each image is horizontally and verticallydownsampled. Since two images are being downsampled, a total of fourdownsampled images are generated. Each of the four downsampled images isthen high pass filtered in the horizontal direction. After high passfiltering, the displacement in the dominant direction is estimated usingthe appropriate pair of downsampled images. For example, if thehorizontal displacements are thought to be larger than the verticaldisplacements within their respective ranges, then the horizontaldisplacement is estimated first using the vertically downsampled images.Note it is impossible to be sure if the horizontal displacement arelarger since the displacements have not been calculated yet, however ina horizontal panorama, the horizontal displacement are naturally largerthan the vertical displacement because it is easy to keep the cameramore or less level while it is more difficult to make a mistake on thehorizontal positioning of the camera for each shot. At this point, inone embodiment, the assumption is that the vertical displacement iszero. This is possible because (a) the vertical displacement is assumedto be small, and (b) the horizontal displacement estimation is carriedout on vertically downsampled images. After the horizontal displacementis obtained, the vertical displacement is estimated using thehorizontally downsampled images. In this case, we take intoconsideration the previously estimated horizontal displacement bytranslating the horizontally downsampled images appropriately.

a. Horizontal and Vertical Downsampling

FIG. 34 is a block diagram of the Displacement Estimation architectureof FIG. 21, according to the present invention. The horizontaldownsampled result is also automatically mirror reflected so that avertical displacement of the original image is equivalent to ahorizontal displacement of the mirror reflected, horizontallydownsampled image. During Horizontal and Vertical Downsampling, multiplevertical scanlines and multiple horizontal scanlines respectively, are(block) averaged. During Horizontal Downsampling, the low-pass result isalso reflected about y=x; that is, each downsampled vertical scanline isstored as a horizontal scanline. This is done so that the sameone-dimensional High Pass Filtering and one-dimensional DisplacementComputation algorithm could be used. Both images are downsampled in thesame way; this step could be done in parallel. The output of this block3400 is the displacement along the horizontal and vertical direction ofthe two pictures. This results are stored into the Register 512. TheRegister 512 is a memory element.

b. One-Dimensional High-Pass Filtering

One-Dimensional High Pass Filtering is performed next on each of thedownsampled images (a total of four: two per image). This high passfiltering is needed to make the error measure robust (invariant) toglobal illumination changes. One-dimensional high pass filtering iscarried out in the horizontal direction only. Since the horizontallydownsampled result has been mirror reflected, one-dimensional high passfiltering in the horizontal direction is the same as filtering in thevertical direction of the unreflected result. The current choice offilter coefficients is [−1, 2, −1].

c. Estimating the Horizontal and Vertical Displacements

The motion estimation required for the digital camera 100 is larger inone direction than the other. When the Panorama is horizontal, thehorizontal displacement is assumed to be larger than the displacement inthe vertical direction. Conversely, when the Panorama is vertical, thevertical displacement is assumed to be larger than the displacement inthe horizontal direction. Thus, and because of the downsampling, we canfirst compute the displacement in the dominant direction, assuming thatthe displacement in the other direction is zero. After this displacementis determined, the displacement in the minor direction is computedtaking into consideration the displacement in the dominant direction.Nevertheless, displacements are always calculated in the horizontaldirection because of the prior rotation of the horizontally downsampledscanlines.

C.2. Calculating the SAD Between Images

FIG. 36 is an illustration of the computation of the sum-of-absolutedifferences (SAD) between a pair of images. The global SAD is computedby accumulating the local SAD between corresponding pairs of scanlines,over all scanlines. In order to determine the optimum horizontaldisplacement between two images, the sum-of-absolute differences (SAD)between the pair of images is computed at every possible displacementwithin the search range. Instead of computing the global SAD between twoimages repeatedly, the local SAD (the SAD between corresponding pairs ofscanlines) is computed for each pair of scanline and accumulated in theSAD Accumulator as illustrated in FIG. 36. This allows the images to beaccessed once. The pseudo-code for the operation is shown below.

For each displacement dx:

SAD[dx]=0.

For each scanline y:

For each displacement dx:

SAD[dx]+=ScanlineSAD(dx, Image1Scanline, Image2Scanline).

Note that the ScanlineSAD function computes the SAD between pairs ofscanline only over the overlap region. Since the overlap region is ofvarying size, the scanline SAD is also normalized; specifically, theSAD/pixel is computed. Alternatively, if the window over which the SADis computed by ScanlineSAD is held fixed, then normalization isunnecessary. Note that the ScanlineSAD for all the displacements withinthe search range could be computed in parallel.

Although SAD is shown, it is important to note that other methods existsto estimate the difference between the two pairs of images. For example,correlation is another method, although more computationally intensive,that can be used in place of SAD. Still other methods exists toaccomplish the same results as SAD know to those of average skill in theart.

C.3. Determining the Optimal Displacement

FIG. 37 is an illustration of the SAD (error) profile. The optimumdisplacement is the displacement corresponding to the minimum localminima in the profile. After the SAD between images has been computed,the SAD accumulator is searched for the minimum local minima. Thedisplacement corresponding to this local minima is the optimumdisplacement. Note that the minimum locall minima is used rather thanthe local minima. In effect, this excludes the minima that could befound on the boundary of the search range as these would not correspondto the artifacts produced of the local search method.

The required operations are implemented by the Displacement Estimationblock depicted in FIG. 34.

If the pictures have to be stitched along the horizontal direction, thenthe displacement along this direction (the dominant one) is computed.The vertical estimation will be computed as a second step. On the otherhand, if the dominant direction is the vertical one, then thedisplacement along this direction will be computed first. The assumptionin this embodiment is that the dominant direction is the horizontal one,however it is important to point out again that all the teachings in thecurrent invention apply to vertical panorama image capturing as well.The downsample blocks compute the two sub-sampled pictures along thevertical direction and store the result in the respective buffers. Thenthese pictures are preprocessed (for ex. an high pass filter is appliedto them) and the Sum of the absolute differences is computed. Furtherdetails on this algorithm are described in the section below entitledAlgorithm for Image Compositing.

D.1. Color Correction Architecture

FIG. 38 is a block diagram 3800 of the Color Correction architecture ofFIG. 21 according to the present invention. The Color CorrectionParameters Estimation architecture is described in FIG. 39. The outputsof these blocks are simple sets of values and these values will be usedin the color Correction block. The values extracted form the current andprevious pictures are called: BrightnessCurrent (Bc), ContrastCurrent(cc), BrightnessPrevious(Bp), ContrastPrevious (Cp). These abbreviationsare used in FIGS. 38 and 40. These values can be stored in any generalpurpose register (not shown).

D.2. Algorithm for Color Correction Estimation

The color correction model that is adopted in the Camera 100 involvesbrightness and contrast histogram matching. Other more sophisticatedtechniques exist but histogram matching was chosen because of itsindependence on motion estimation results and could therefore becomputed in parallel.

Denote I_prev as a color channel of the previous buffer (prevBuffer1802), and I_cur as the same color channel of the Current Buffer(curBuffer 1808). Assume a default overlap between the previous andCurrent Buffers; this default could be the default position of theCurrent Buffer with respect to the previous buffer. Within the area ofoverlaps in the two images, compute:

E _(—) prev=1/N Sum(I _(—) prev); E 2 _(—) prev=1/N Sum(I _(—) prev×I_(—) prev)

Var _(—) prev=E 2 _(—) prev−(E _(—) prev×E _(—) prev)

E _(—) cur=1/N Sum(I _(—) cur); E 2 _(—) cur=1/N Sum(I _(—) cur×I _(—)cur)

Var _(—) cur=E 2 _(—) cur−(E _(—) cur×E _(—) cur)

The brightness and contrast to apply to the Current Buffer to match theprevious buffer is given as:

Brightness=E_prev−Contrast×E_cur

Contrast=sqrt(Var_prev/Var_cur)

Instead of color correcting the Current Buffer only, we apply colorcorrection to both buffers equally using the following equations:

ΔI=(Brightness+Contrast×I)−I

I′=I+β×ΔI

Color correction is later applied according to the following equations:

I′ _(—) prev=Brightness_(—) prev+Contrast_(—) prev×I _(—) prev

I′ _(—) cur=Brightness_(—) cur+Contrast_(—) cur×I _(—) cur

This color correction technique is applied to each color channelindependently. For example, if the buffers are YCC, separate brightnessand contrast parameters for each of the channel is calculated.

D.3. Algorithm for Color Correction

In this section, a description of the color correction that is appliedto the previous and Current Buffers. The parameters of the colorcorrection are the respective brightness and contrast adjustments foreach color channel that was computed earlier by matching the buffers′histograms.

Turning now to FIG. 40, shown is an illustration of the amount of colorcorrection applied at various regions of the previous and CurrentBuffer.

Color correction is applied to varying degrees in different parts ofeach buffer. Within the region of overlap, color correction is appliedfully (β=1); outside of the region of overlap, color correction isapplied less further away from the overlap (0<=β<1).

Brightness_prev=Brightness/sqrt(Contrast)

Contrast_prev=1/sqrt(Contrast)

Brightness_cur=Brightness/sqrt(Contrast)

Contrast_cur=sqrt(Contrast).

A variant of this scheme does not apply full color correction evenwithin the overlapping region. Instead, only up to 75% of colorcorrection is ever applied. Thus, beta ranges from 0 to 0.75 outside ofthe overlapping region, and is 0.75 within. Using this technique, theprevious and Current Buffers are not sufficiently color corrected tomatch each other. This is acceptable because the subsequent imagecompositing step transitions smoothly the previous buffer into theCurrent Buffer. Not completely color correcting the two buffers isuseful as it produces a smoother color transition between the twobuffers.

E.1. Image Splicing Architecture

FIG. 40 is a block diagram of the Stitching and Blending architecture ofFIG. 21, according to the present invention.

E.2. Algorithm for Image Compositing

FIG. 41 is an illustration showing the regions of the previous andCurrent Buffer during image composition, according to the presentinvention. This section describes the image compositing between previousand Current Buffer that is carried out to construct the seamlessPanorama. Since the region of the previous buffer between prevFarEdgeand prevNearEdge will be written out next, the composited result isupdated onto the previous buffer.

Only the overlap region between the previous and Current Buffers arecomposited. The two buffers are composited according to the followingequation:

I′ _(—) prev=α*I _(—) prev+(1−α)*I _(—) cur

where a is a weighting factor that varies linearly between one and zeroas shown in FIG. 41.

E.3. Algorithm for Image Splicing

This section describes a modification of the image compositing techniquedescribed earlier. During image compositing, the previous and CurrentBuffers are alpha-blended within their region of overlap. While thisresults in a smooth transition between the two buffers, it producesghosting when the content in the two buffers differ. For example, if aperson is in the overlap region within one buffer and not the other, aghost-like reproduction of that person will be present in the finalPanorama. If the compositing takes place over a narrower region of theoverlap region to the left or to the right of the person, then theresulting.Panorama will either contain the person entirely or notinclude the person (respectively). FIG. 42 shows an illustration of twobuffers being composited over a narrow region.

Turning now to FIG. 42, shown is illustration of the previous andCurrent Buffer being composited over a narrower region. The location ofthe center of the narrower compositing region could be taken as thelocation that yields the least SAD error of a given error function. Onesuch error function would be the sum of absolute differences across allcorresponding pixels of each vertical column within the overlappingregion. This would be computed with the Current Buffer at its optimumdisplacement with respect to the previous buffer. The width of thecompositing region could be fixed or determined from the same errorfunction. For example, if the minima of the error function (with whichwe determine the location) is narrow, then the width should be narrow.Conversely, if the minima is wide, then the width of the compositingregion should be larger.

FIG. 43 illustrates the regions of the previous and Current Buffercontributing to the final Panorama. Note that when compositing over thenarrower compositing region, the prevNearEdge is no longer the rightedge of the previous buffer. Instead, it is set to be the right edge ofthe compositing region. Thus, only the region of the previous bufferbetween the prevFarEdge and the prevNearEdge is saved out.

F.1. Algorithm for Cylindrical to Rectilinear Transform

Turning now to FIG. 44, shown is an illustration of the Current Viewwithin the Panorama. The Cylindrical to Rectilinear Transform transformsa portion of a cylindrical Panorama into rectilinear coordinages. Thistransform is basically the inverse of the Rectilinear to CylindricalTransform described in a previous section. The panoramic viewer renderseach new view of the Panorama using this transform. Each new view isparameterized by the origin in cylindrical coordinates (θ_origin), thefocal length (f), and the dimensions of the viewport. Panning thepanoramic viewer involves changing the θ_origin parameter andre-rendering the new view.

Like the Rectilinear to Cylindrical Transform, the Cylindrical toRectilinear Transform uniformly rescales each vertical scanline in theimage independently. Thus, the transform is described by: (a) thelocation of the source scanline corresponding to each destinationscanline, and (b) the scale factor for each scanline.

F.2. Calculating the X-coordinate Transformation

θ_(—) src=a tan2(x _(—) dst, f)+θ_origin

The location of each source vertical scanline (q_src) depends on thedestination scanline (x_dst), the focal length (f), and the origin(θ_origin).

F.3. Calculating Y-scanline Rescaling

y _(—) src=yscale*y _(—) dst

Each vertical scanline is a uniformly rescaled copy of the original. Thescale factor is unnormalized by the factor y′_scale_min that was appliedin the forward transform.

y′_scale=f/sqrt(tan²θ_(—) src+1)=f ²/sqrt(x _(—) dst ² +1)

y′_scale_min=min(f/sqrt(tan²θ_min+1), f/sqrt(tan²θ_max+1))

y_scale=y′_scale_min_y′_scale

Thus, the scale factor depends on the location of the destinationscanline (and ultimately, the location of the source scanline), and thefocal length. The denominator of y′_scale can be precomputed as alook-up table, indexed by the location of the destination scanline(x_dst).

6. Motion Play-Back of Still Pictures That Have Been Previously Stored

The digital camera has previously stored images such as images A, B, C,D of FIG. 3 that have been stitched together as described above. Thefollowing section describes how these previously stored images can bedisplayed with a perspective and motion of a video camera that couldhave captured the images comprising the panoramic scene. It is assumedthat these images are available. This process of motion play-back (MPB)is based upon the continuously scanning of the picture in order tosimulate the motion of a video camera acquiring the scene that isdisplayed.

Turning now to FIG. 45, shown is a high level architecture of a system4500 for implementing motion play-back (MPB), according to the presentinvention. An Image Generator 4502 contains all the resources (H/W andS/W) to: (1) create the picture by stitching several pictures togetherand then transfer it to the Memory 4504; or (2) up-load a picture from anon volatile memory, such as storage media 142, decode it and send it tothe memory 4504. The memory 4504 is a standard block that contains thepictures. The memory 4504 is connected to a motion play-back (MPB) 4506by address 4510 and data lines 4512. The MPB 4506 generates theaddresses to properly read the picture in order to simulate the motionaccording to the parameters provided with the settings 451A. Thesettings 4514 provides information related to:

Speed of motion.

Frame Frequency.

Raster Structure: Interlaced/Progressive.

ViewPort Horizontal and Vertical (Xv-Yv) dimension. Shown is a viewport(the rectangle with θ_origina at is venter where Xv and Yv are theviewport dimensions.

Picture Horizontal and Vertical (Xv-Yv) dimension.

Source Picture Reference System: Cylindrical, Planar, Spherical.

Picture Starting address.

Focal Length (f) of the optic used to take the picture.

The picture information is provided to the MPB 4506 by the data line4512 and the MPB 4506 processes them according to the settings 4514.

Now an example is provided in order to describe how the MPB 4514 works.FIG. 46 depicts a picture having the horizontal extent, i.e. along the Xdirection, equals 2 times the horizontal extent of the viewport. It isassumed the viewport is an LCD 126 or output, such as PAL or NTSC, to aTV set with interlaced scanning. Interlace is illustrated as even rows4062 and odd rows 4064. During the first field (the odd one, forexample) the portion of the picture to be displayed is depicted in theleft-most part of FIG. 46 in window 4606. For the even field (the nextone) the portion to be displayed is shifted by an offset 4608 to showthe following window 4610. This process is repeated so that the viewport(i.e. window 466) is repeatedly moved by offset 4608 to form subsequentview 5 as desired. The actual interfaced even rows 4604 that aredisplayed for viewing window 466. Similarly, for viewing window 4610,the odd rows 4604 are displayed.

The MPB is described now based on the following assumptions: (1) RasterStructure: Interlaced; and (2) Source Picture Reference System:Cylindrical. FIG. 47 is an illustration of the coordinate systems 4700used for the stored pictures 4702 and the raster display pictures 4704,according to the present invention. The stored images 4702 are thepreviously stitched frames. In order to properly display a picture, thecylindrical-to-planar conversion must be implemented. Generally, thisgraphic transformation is applied in inverse mode, i.e. starting fromthe position of the destination picture (corresponding to the positionof a point of the display, in this case) the corresponding point in thesource picture is found. The equations describing thecylindrical-to-planar mapping applied in the inverse mode are:

Xs=arctan(X/f)+origin  (1)

Ys=Y_scale×Yd  (2)

Y _(—scale=sqr)(f)/sqrt(sqr(Xd)+1)  (3)

Depending on the speed of the motion to be simulated, an offset 468 mustbe added to the address related to the pixel position along thehorizontal direction of source picture. This offset 4608 depends on theRaster structure. For example if the raster is interlaced this offset4608 must be updated for every field. On the other hand, if the rasteris progressive, this Offset must be updated every frame. Where frepresents the focal length. Equations (1), (2) and (3) above can beimplemented in a variety of hardware and software combinations,including implementing the mathematical functions with a look up table,but this is not restrictive for the invention. FIG. 48 is a blockdiagram 4800 of the major components of MPB 4506 of FIG. 45, accordingto the present invention. A counter 4802 generates the coordinate of apoint in the display coordinates 4704. The FP, 4818 signal directs whenthe offset 4608 must be updated. FIG. 49 is an illustration of theX-mapping 4808 for carrying out equation (1). FIG. 50 is an illustrationof the Y-mapping 4806 for carrying out equations (2) and (3). AddressMapping 4810 translates the cylindrical coordinates Xs and Ys of 4702 toplanar coordinates Xd and Yd 4704.

Other embodiments are possible. For example, if the input picture is inplanar coordinates, equations (1), (2) and (3) do not implement anytransformation in the MPB 4506. On the other hand if the input picturein spherical coordinates the MPB 4506 implements a different set oftransforms. The physical address of a picture coordinate is used toretrieve a pixel from the memory. Due to the nature of the transform, itis possible the location Xs and Ys does not correspond to a pixelalready existing on the source image. This location can be between topixels. To recover this drawback, the Pixel Processor block implementsan interpolation algorithm in order to compute the pixel value at theposition Xs, Ys. Moreover the Picture Processing 4812 implements thedown-sampling along the vertical direction in order to fit the verticalsize of the viewport.

One architecture of the Picture Processing 4812 is depicted in FIG. 51.And one architecture of the Vertical Sub-Sampling 5102 is depicted inFIG. 52. The vertical Sub-Sampling 5102 requires some Line Memories (LM)and Pixel Delays (PD) in order to implement the processing windowrequired by the particular Interpolation algorithm implemented by theSS_Interp block. Some of those algorithms require the Field/Frame Signaltoo. The Interpolation block, computes the value of the pixel at the XsYs position starting from the available ones in the source picture. Apossible implementation of the Interpolation 5104 using a CubicInterpolator is described in FIG. 53 while FIG. 54 illustrates apossible implementation of the Interpolation block using the linearinterpolation algorithm.

Although a specific embodiment of the invention has been disclosed, itwill be understood by those having skill in the art that changes can bemade to this specific embodiment without departing from the spirit andscope of the invention. The scope of the invention is not to berestricted, therefore, to the specific embodiment, and it is intendedthat the appended claims cover any and all such applications,modifications, and embodiments within the scope of the presentinvention.

What is claimed is:
 1. A method for converting a rectilinear image and afocal length into a cylindrical image parameterized by a height of thecylinder and an angular distance in a single buffer on a remoteprocessing device, the method comprising the steps of: collecting asource rectilinear image in a single buffer from a camera; dividing thesingle buffer with a first end and a second end, into a series ofsuccessive scan lines of predetermined width, the single buffercontaining the source rectilinear image; grouping the series of scanslines, into two groups, a first group of adjacent scan lines having afirst boundary at the first end of the buffer and a second boundary at apredetermined position of an angular distance, and a second group ofscan lines defined by the remainder of the scan lines within the singlebuffer not within the first group, transforming each scan line dividedinto the first and second group from rectilinear coordinates tocylindrical coordinates comprising the successive sub-steps of: (a)reading each source scan; (a.1) wherein each successive source scan linein the first group is read from the buffer at a position starting fromthe first end and continuing to the predetermined angular position;(a.2) wherein each source scan line in the second group is read from thebuffer at a position starting from the second end and continuing to thepredetermined angular position; (b) transforming each scan line readfrom rectilinear coordinates to cylindrical coordinates; and (c) writingeach scan line transformed back into the buffer at the position in thebuffer from where the source scan line was read; and (d) repeating thesub-steps (a) through (e) above until all the scan lines have beentransformed.
 2. The method according to claim 1, wherein the sub-steps(a) through (c) are performed in parallel for each source scan line fromthe first group and each source line from a second group.
 3. The methodaccording to claim 1, further including the steps for estimating themotion between a first destination image of cylindrical coordinates anda second destination image of cylindrical coordinates comprising thesteps of: downsampling a first image in a first direction and in asecond direction; downsampling a second image in the first direction andin the second direction; filtering the first and the second image so asto filter out any global illumination changes between the first imageand the second image; calculating a first displacement along the firstdirection between the first downsampled image along the first directionand the second downsampled image along the first direction; andcalculating a second displacement along the second direction between thefirst downsampled image along the second direction and the seconddownsampled image along the second direction.
 4. The method according toclaim 3, further the comprising the step of: converting a pair of imagesfrom rectangular coordinates to cylindrical coordinates to create afirst image and a second image.
 5. The method according to claim 3,wherein the step of calculating a first displacement includescalculating a first displacement based on the displacement along thesecond direction being zero.
 6. The method according to claim 3, whereinthe step of calculating a first displacement includes the sub-step of:accumulating the sum-of-absolute-differences (SAD) between a predefinedarea of the first image and a predefined area of the second image. 7.The method according to claim 1, further comprising the steps forcorrecting color between a first destination image of cylindricalcoordinates and a second destination image of cylindrical coordinatescomprising the steps of: receiving a color channel from at least thefirst image and the second image; creating an overlap portion betweenthe first image and second image; and adjusting the color channel forthe first image and for the second image in at least the overlap portionbetween the first image and the second image which is independent ofmotion estimation.
 8. The method as according to claim 7, wherein thestep of adjusting the color further comprises the sub-step of: computingthe brightness (B₁) and contrast (C₁) for the first color channel;computing the brightness (B₂) and contrast (C₂) for the second colorchannel; adjusting color correction in at least the overlap portionbetween the first image and the second image; setting the color channelfor the first image is set equal to: I₁=B₁+C₁×I₁; and setting the colorchannel for the second image is set equal to I₂=B₂+C₂×I₂.
 9. The methodaccording to claim 8, wherein the step of adjusting the color furthercomprises the sub-steps of: computing a histogram of color distributionfor the first color channel (H₁); computing a histogram of colordistribution for the second color channel (H₂); and setting B₁ and B₂equal to: H₂−(matched contrast (C))×H₁ wherein the C is set equal to thesquare root of a variance calculated for H₁ divided by a variancecalculated for H₂.
 10. A method for converting a rectilinear imageparameterized by an optical center (x_ctr, y_ctr) and a focal length (f)into a cylindrical image parameterized by a height (h) and an angulardistance (_) in a single buffer on a remote processing device, themethod comprising the steps of: collecting a source rectilinear image ina single butter from a camera; dividing the single buffer with a firstend and a second end, into a series of successive scan lines ofpredetermined width, the single buffer containing the source rectilinearimage; grouping the series of scans lines, into two groups, a firstgroup of adjacent scan lines having a first boundary at the first end ofthe buffer and a second boundary at a predetermined position of anangular distance, and a second group of scan lines defined by theremainder of the scan lines within the single buffer not within thefirst group; reading a source scan line at a location (col_src) on therectilinear source image from the single buffer so thatcol_src=x_src+x_ctr; transforming the source rectilinear scan line(x_src, y_src) into a destination cylindrical scan line (x_dst, y_dst)based on: a location of each vertical scanline as defined by theequation: col _(—) src=col _(—) dst+(col _(—) src−col _(—) dst);  wherecol_src=f*tan _+x_ctr; and col_dst is a destination column location inthe buffer; a scale factor for each source scan line (y_src) defined bythe equation: y _(—) src=[(f/sqrt(tan²_(—)+1))]/[(min(f/sqrt(tan²_min+1), f/sqrt(tan²_max+1)]* y _(—) dst; where _min=a tan2(−x_src, f); _max=a tan2(width-x_src, f); width is thewidth of the source rectilinear image; and writing the cylindrical scanline at a location (col_dst) in the buffer.
 11. A computer readablemedium containing programming instructions for converting a rectilinearimage and a focal length into a cylindrical image parameterized by aheight of the cylinder and an angular distance in a single buffer on aremote processing device, the programming instructions comprising:collecting a source rectilinear image in a single buffer from a camera;dividing the single buffer with a first end and a second end, into aseries of successive scan lines of predetermined width, the singlebuffer containing the source rectilinear image; grouping the series ofscans lines, into two groups, a first group of adjacent scan lineshaving a first boundary at the first end of the buffer and a secondboundary at a predetermined position of an angular distance, and asecond group of scan lines defined by the remainder of the scan lineswithin the single buffer not within the first group; transforming eachscan line divided into the first and second group from rectilinearcoordinates to cylindrical coordinates comprising the successivesub-steps of: (a) reading each source scan; (a.1) wherein eachsuccessive source scan line in the first group is read from the bufferat a position starting from the first end and continuing to thepredetermined angular position; (a.2) wherein each source scan line inthe second group is read from the buffer at a position starting from thesecond end and continuing to the predetermined angular position; (b)transforming each scan line read from rectilinear coordinates tocylindrical coordinates; and (c) writing each scan line transformed backinto the buffer at the position in the buffer from where the source scanline was read; and (d) repeating the sub-steps (a) through (c) aboveuntil all the scan lines have been transformed.
 12. The computerreadable medium according to claim 11, wherein the programminginstructions (a) through (c) are performed in parallel for each sourcescan line from the first group and each source line from a second group.13. The computer readable medium according to claim 11, furtherincluding the programming instructions for estimating the motion betweena first destination image of cylindrical coordinates and a seconddestination image of cylindrical coordinates comprising the programminginstructions of: downsampling a first image in a first direction and ina second direction; downsampling a second image in the first directionand in the second direction; filtering the first and the second image soas to filter out any global illumination changes between the first imageand the second image; calculating a first displacement along the firstdirection between the first downsampled image along the first directionand the second downsampled image along the first direction; andcalculating a second displacement along the second direction between thefirst downsampled image along the second direction and the seconddownsampled image along the second direction.
 14. The computer readablemedium according to claim 13, further the comprising the programminginstructions of: converting a pair of images from rectangularcoordinates to cylindrical coordinates to create a first image and asecond image.
 15. The computer readable medium according to claim 13,wherein the programming instruction of calculating a first displacementincludes calculating a first displacement based on the displacementalong the second direction being zero.
 16. The computer readable mediumaccording to claim 13, wherein the programming instruction ofcalculating a first displacement includes the programming instructionof: accumulating the sum-of-absolute-differences (SAD) between apredefined area of the first image and a predefined area of the secondimage.
 17. The computer readable medium according to claim 11, furthercomprising the programming instructions for correcting color between afirst destination image of cylindrical coordinates and a seconddestination image of cylindrical coordinates comprising the instructionsof: receiving a color channel from at least the first image and thesecond image; creating an overlap portion between the first image andsecond image; and adjusting the color channel for the first image andfor the second image in at least the overlap portion between the firstimage and the second image which is independent of motion estimation.18. The computer readable medium according to claim 17, wherein theprogramming instruction of adjusting the color further comprises thesub-step of: computing the brightness (B₁) and contrast (C₁) for thefirst color channel; computing the brightness (B₂) and contrast (C₂) forthe second color channel; adjusting color correction in at least theoverlap portion between the first image and the second image; settingthe color channel for the first image is set equal to: I₁=B₁+C₁×I₁; andsetting the color channel for the second image is set equal toI₂=B₂+C₂×I₂.
 19. The computer readable medium according to claim 18,wherein the programming instruction of adjusting the color furthercomprises the sub-steps of: computing a histogram of color distributionfor the first color channel (H₁); computing a histogram of colordistribution for the second color channel (H₂); arid setting B₁ and B₂equal to: H₂−(matched contrast (C))×H₁ wherein the C is set equal to thesquare root of a variance calculated for H₁ divided by a variancecalculated for H₂.